Textual Properties and Task-based Evaluation: Investigating the Role of Surface Properties, Structure and Content

This paper investigates the relationship between the results of an extrinsic, taskbased evaluation of an NLG system and various metrics measuring both surface and deep semantic textual properties, including relevance. The latter rely heavily on domain knowledge. We show that they correlate systematically with some measures of performance. The core argument of this paper is that more domain knowledge-based metrics shed more light on the relationship between deep semantic properties of a text and task performance.